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Abstract — The optimal prefix-free machine U is a universal 
decoding algorithm used to define the notion of program-size 
complexity H(s) for a finite binary string s. Since the set 
of all halting inputs for U is chosen to form a prefix-free 
set, the optimal prefix-free machine U can be regarded as 
an instantaneous code for noiseless source coding scheme. In 
this paper, we investigate the properties of optimal prefix-free 
machines as instantaneous codes. In particular, we investigate 
the properties of the set U~ 1 (s) of codewords associated with 
a symbol s. Namely, we investigate the number of codewords in 
U~ 1 (s) and the distribution of codewords in U~ 1 (s) for each 
symbol s, using the toolkit of algorithmic information theory. 

I. Introduction 

Algorithmic information theory (AIT, for short) is a frame- 
work for applying information-theoretic and probabilistic ideas 
to recursive function theory. One of the primary concepts of 
AIT is the program-size complexity (or Kolmogorov complex- 
ity) H(s) of a finite binary string s, which is defined as the 
length of the shortest binary input for a universal decoding 
algorithm U, called an optimal prefix-free machine, to output 
s. By the definition, H (s) can be thought of as the information 
content of the individual finite binary string s. In fact, AIT has 
precisely the formal properties of normal information theory 
(see Chaitin [1]). On the other hand, H(s) can also be thought 
to represent the amount of randomness contained in a finite 
binary string s, which cannot be captured in a computational 
manner. In particular, the notion of program-size complexity 
plays a crucial role in characterizing the randomness of an 
infinite binary string, or equivalently, a real. 

The optimal prefix-free machine U is chosen so as to satisfy 
that the set dom U of all halting inputs for U forms a prefix- 
free set. Therefore, as considered in Chaitin [1|, we can think 
of the optimal prefix-free machine C/asa decoding equipment 
at the receiving end of a noiseless binary communication 
channel. We can regard its programs (i.e., finite binary strings 
in domU) as codewords and can regard the result of the 
computation by U, which is a finite binary string, as a decoded 
"symbol." Since domU is a prefix-free set, such codewords 
form what is called an "instantaneous code," so that successive 
symbols sent through the channel in the form of concatenation 
of codewords can be separated!] 

Thus, from the point of view of information theory, it is 
important to investigate the properties of optimal prefix-free 

1 Note that AIT does not assume the existence of an encoding algorithm E 
such that E(s) = p if and only if U(p) = s. 



machine as an instantaneous code. In this paper, in particular, 
we investigate the properties of the set U~ 1 (s) of codewords 
associated with a symbol s, where U~ 1 (s) = {p \ U(p) — s }. 
Unlike for instantaneous codes in normal information theory, 
the codeword p associated with each symbol s by s ~ U(p) 
is not necessarily unique for optimal prefix-free machines U 
in AIT. We investigate this property from various aspects. 

After the preliminary section, in Section [III] we investigate 
the number of codewords in U~ 1 (s). We show the following: 
(i) While keeping H(s) unchanged for all s, we can modify 
U so that each U~ 1 (s) is a finite set, where the number of 
codewords in U (s) is bounded to the above by some total 
recursive function /(s), i.e., by some computable function 
/(s). (ii) This upper bound f(s) cannot be chosen to be tight 
at all. (iii) As a result, even in the case where all U~ 1 (s) are a 
finite set, the number of codewords in U~ 1 (s) is not bounded 
to the above on all finite binary strings s. (iv) While keeping 
H(s) unchanged for all s, we can modify U so that each 
U~ 1 (s) is an infinite set. In Section [IV] we then investigate 
the distribution of codewords in U~ 1 (s). We estimate the 
distribution using the notion of program-size complexity, and 
then show that the estimation is tight. 



II. Preliminaries 



A. Basic Notation 



We start with some notation about numbers and strings 
which will be used in this paper. #5 is the cardinality of 
S for any set S. N = {0,1,2,3,...} is the set of natural 
numbers, and N + is the set of positive integers. Q is the set 
of rationals, and R is the set of reals. Normally, O(l) denotes 
any function / : N+ -> R such that there isCel with the 
property that \f(n)\ < C for all n G N+. 

{0,1}* = {A, 0,1, 00, 01, 10, 11, 000,...} is the set of 
finite binary strings where A denotes the empty string, and 
{0, 1}* is ordered as indicated. We identify any string in 
{0, 1}* with a natural number in this order, i.e., we consider 
ip: {0,1}* — > N such that <p(s) = Is — 1 where the 
concatenation Is of strings 1 and s is regarded as a dyadic 
integer, and then we identify s with ip(s). For any s G {0, 1}*, 
\s\ is the length of s. A subset S of {0, 1}* is called prefix- 
free if no string in S is a prefix of another string in S. For 
any function /, the domain of definition of / is denoted by 
dom/. We write "r.e." instead of "recursively enumerable." 



B. Algorithmic Information Theory 

In the following we concisely review some definitions and 
results of AIT Q), El* EL ID- A prefix-free machine is a 
partial recursive function C : {0,1}* —5- {0,1}* such that 
domC is a prefix-free set. For each prefix-free machine C 
and each s g {0, 1}*, H c (s) is defined by 

H c (s) = min { \p\ | p g {0, 1}* & C(p) = s } (may be oo). 

A prefix-free machine U is said to be optimal if for each 
prefix-free machine C there exists dsN with the following 
property; if p g domC, then there is q for which U(q) — C(p) 
and |g| < \p\ +d. Note that a prefix-free machine U is optimal 
if and only if for each prefix-free machine C there exists d g N 
such that, for every s g {0, 1}*, Hj/( 5 ) < #c(s) + d. It is 
easy to see that there exists an optimal prefix-free machine. 
We choose a particular optimal prefix-free machine U as 
the standard one for use, and define H(s) as Hjj(s), which 
is referred to as the program-size complexity of s or the 
Kolmogorov complexity of s. It follows that for every prefix- 
free machine C there exists d G N such that, for every 

se{0,i}*, 

H{s)<H c {s)+d. (1) 

Based on this we can show that, for every partial recursive 
function <£: {0, 1}* -> {0, 1}*, there exists d eN such that, 
for every s g dom \&, 

#(*(*)) < #(s) + d. (2) 

Based on dTJ we can also show that there exists cfN such 
that, for every n g N + , 

F(n) < 21og 2 n + c. (3) 

For any s £ {0,1}*, we define s* as min{p g {0,1}* | 
U(p) = s}, i.e., the first element in the ordered set {0,1}* 
of all strings p such that U(p) = s. Then, \s*\ = H(s) for 
every s g {0, 1}*. For any s, t g {0, 1}*, we define il(s, i) as 
H(b(s,t)), where 6: {0, l}*x{0, 1}* ->■ {0, 1}* is a particular 
bijective total recursive function. 

AIT has precisely the formal properties of normal infor- 
mation theory, as demonstrated by Chaitin [ 1 1 . The program- 
size complexity H(s) corresponds to the notion of entropy in 
information theory, while H(s, t) corresponds to the notion of 
joint entropy in information theory. 

The program-size complexity H(s) is originally defined 
using the notion of program-size, as in the above. However, 
it is possible to define H(s) without referring to such a 
notion. Namely, as in the following, we first introduce a 
universal probability m, and then define H(s) as — log 2 m(s). 
A universal probability is defined as follows. 

Definition 1 (universal probability, Zvonkin and Levin [8|). 
A function r: {0,1}* —¥ [0,1] is called a lower-computable 
semi-measure if X) s pfo 11* r ( s ) — ^ an< ^ f ^ e set i( a ' s ) ^ 
Q x {0, 1}* | a < r(s)} is r.e. We say that a lower-computable 
semi-measure m is a universal probability if for every lower- 
computable semi-measure r, there exists c G N + such that, for 
all s £ {0,1}*, r(s) < cm(s). ■ 



The following theorem can be then shown (see e.g. Chaitin 
|fl] Theorem 3.4] for its proof). 

Theorem 2. For every optimal prefix-free machine V, the 
function 2~ Hv ^ s > of s is a universal probability. ■ 

For each universal probability m, by Theorem [2] we see 
that H(s) = - log 2 m(s) + 0(1) for all s G {0, 1}*. Thus 
it is possible to define H(s) as — log 2 m(s) with a particular 
universal probability m instead of as Hjj(s). Note that the 
difference up to an additive constant is nonessential to AIT. 

Normally, for each prefix-free machine C and each s g 
{0, 1}*, the set C^ 1 (s) is defined by 

C-\s) = {pe domC | C(p) = s}. 

Note that V r_1 (s) ^ for every optimal prefix-free machine 
V and every s g {0, 1}*. 

III. The Number of Codewords 

In this section, we investigate the properties of the number 
ifV~ 1 {s) of codewords in V^ 1 (s) for an optimal prefix-free 
machine V. In Theorem |4]below we show that, while keeping 
Hv(s) unchanged for all s, we can modify V so that each 
V^ 1 (s) is a finite set, where #V~ l (s) is bounded to the above 
by some total recursive function f(s). Before that, we prove 
a more general theorem for prefix-free machines in general, 
as follows. 

Theorem 3. For every prefix-free machine C, there exists a 
prefix-free machine D for which the following conditions (i), 
(ii), and (Hi) hold: 

(i) H D (s) = H c (s) for every s g {0, 1}*. 
(ii) D -1 ^) is a finite set for every s g {0, 1}*. 
(iii) Moreover, there exists a partial recursive function 
f: {0, 1}* -> N+ such that #Z?" 1 ( S ) < f( s ) f° r eve T 
sedomf and dom f = {se {0,1}* | D-\s) ^0}. 

Proof: Let C be an arbitrary prefix-free machine. We 
define the graph Graph(C) of C by 

Graph(C) = { (p, s) g {0, 1}* x {0, 1}* | C(p) = s }. 

Note that Graph(C) is an r.e. set, since C : {0, 1}* -)• {0, 1}* 
is a partial recursive function. In the case where Graph(C) is 
a finite set, the set { s g {0, 1}* | C _1 ( s ) ^ } is finite and 
the set C^ 1 (s) is finite for every s g {0, 1}*. Thus, in this 
case, by setting D — C and f(s) = ^C^ 1 (s), the conditions 
(i), (ii), and (iii) hold, and therefore the result follows. Hence, 
in what follows we assume that Graph(C) is an infinite set. 
Let (pi, si), (p 2 , s 2 ), (j>3, S3), . . . be a particular recursive 
enumeration of the infinite r.e. set Graph(C). It is then easy to 
show that there exists a partial recursive function g : {0, 1}* — > 
N + which satisfies the following two conditions: 

(a) dom g = { a | 3i g N + Si = s }. 

(b) g(s) = rnin{ i g N + | Si — s } for every s g domg. 
We then define a partial recursive function D: {0,1}* — > 
{0, 1}* by the condition that 

D-\s) = { Pi J i g N+ & Si = s & \ Pi \ < \ Pg(s) \} 



if s £ doing and D^ 1 (s) = otherwise. It is easy to see 
that such a partial recursive function D exists. By counting 
the number of binary strings of length at most |p 9 ( s )|> we 
see that, for each s € doing, #£>~ 1 (s) < 2l p s(=)l +1 - 1 
and therefore D^ 1 (s) is a finite set. Thus, the condition (ii) 
holds for D. Moreover, by defining a partial recursive function 
/: {0, 1}* — > N + by the conditions that dom/ = doing and 
f(s) = 2l p 9< a 'l +1 — 1 for every s £ dom/, the condition (iii) 
holds for D. 

Next, we show that D is a prefix-free machine. It follows 
from the definition of D that 

D-'^cC-Hs) (4) 

for every s £ {0, 1}*. Therefore we see that 

domD= (J D~ 1 (s)C (J C" 1 (s)=domC. 
se{o,i}* se{o,i}* 

Thus, since domC is prefix-free, its subset dom I? is also 
prefix-free. Hence I? is a prefix-free machine. 

Finally, we show that the condition (i) holds for D. Let 
us assume that C(p) — s and \p\ = Hc(s). Then (p,s) £ 
Graph(C) and therefore s £ domg. Since C(p g r a \) = s, 
we see that \p\ < \p g < s )\ and therefore p £ D^ 1 (s). Hence, 
D(p) = s and therefore Hd(s) < \p\. Thus we have 

Hd(s)<H c (s) (5) 

for every s £ {0, 1}*. On the other hand, (0]i implies that 

H D (s) > H c (s) (6) 

for every s £ {0, 1}*. It follows from (0) and © that the 
condition (i) holds for D. ■ 

Theorem 4. For every optimal prefix-free machine V, there 
exists an optimal prefix-free machine W for which the follow- 
ing conditions (i), (ii), and (iii) hold: 

(i) H w (s) = H v (s) for every s £ {0, 1}*. 
(ii) W~ 1 (s) is a finite set for every s £ {0, 1}*. 
(iii) Moreover, there exists a total recursive function 
f: {0,1}* ^N+ such that ifW- 1 (s) < J '(s) for every 
s£{0,l}*. 

Proof: Let V be an arbitrary optimal prefix-free machine. 
Then it follows from Theorem [3] that there exists a prefix-free 
machine W for which the following conditions (a), (b), and 
(c) hold: 

(a) H w (s) = H v {s) for every s £ {0, 1}*. 

(b) W^ 1 (s) is a finite set for every s £ {0, 1}*. 

(c) Moreover, there exists a partial recursive function 
/: {0, 1}* -> N+ such that #W _1 (s) < f(s) for every 
sedom/ anddom/ = {se {0,1}* | W^O) ^0}. 

Therefore, the conditions (i) and (ii) hold obviously. Since V 
is optimal, W is also optimal by the above condition (a). On 
the other hand, since W is optimal, W" 1 (s) ^ for every 
s £ {0, 1}*. Thus, the condition (iii) holds. ■ 

Through Theorems [6] and [7] below, we show that the upper 
bound f(s) in Theorem [4] cannot be chosen to be tight at all. 



We first show a weaker result, Theorem [6] Then, based on 
this, we show a stronger result, Theorem [7] The underlying 
idea of the proofs of Theorems [6] and [7] is due to A. R. Meyer 
and D. W. Loveland pp. 525-526] (see also Chaitin CD 
Theorem 5.1 (f)]). In order to prove Theorem |6l we need 
Lemma [5] below. It is a well-known fact and follows from 
the inequality #{ s £ {0, 1}* | H(s) < n} < 2 n - 1. 

Lemma 5. Let R be an infinite subset of {0,1}*. Then the 
function H(s) of s £ R is not bounded to the above. ■ 

A function / : {0, 1}* — > N is called right-computable if the 
set { (s, n) £ {0, 1}* x N | /(s) < n } is r.e. Obviously, every 
total recursive function /:{0,1}*— >Nis right-computable. 

Theorem 6. Let V be an optimal prefix-free machine, and 
let /: {0, 1}* -► N. Suppose that #V _1 (s) < f(s) for all 
S £ {0, 1}* and f is right-computable. Then ^V^ 1 (s) < /(s) 
for all but finitely many s £ {0, 1}*. 

Proof: We define a function h by the following two 
conditions: 

(a) domh = { s £ {0, 1}* | W-\s) - f(s) }. 

(b) h(s) — min{ \p\ \ p £ V^ 1 (s), } for every s £ domh. 

Note first that ^ _1 ( s ) ^ $ f° r every s £ {0,1}* since 
V is optimal. Therefore min{ \p\ \ p £ V^ 1 (s),} is well- 
defined as a natural number for every s £ {0,1}*. Since 
W~\s) < f(s) for all s £ {0,1}* and / is right- 
computable, it is easy to see that the above two conditions 
(a) and (b) define a partial recursive function h: {0, 1}* — > N. 
On the other hand, it follows from the condition (b) that 



h(s) = H(s) 



(7) 



for every s £ dom h. 

Now, let us assume contrarily that #l^ _1 (s) = f(s) for 
infinitely many s £ {0,1}*. Then, obviously, dom h is an 
infinite set. It follows from Lemma |5J the function h is not 
bounded to the above. Thus, given n £ N + , by enumerating 
the graph of the partial recursive function h, one can find 
s £ dom h such that n < h(s). 

Hence, combined with @, we see that there exists a partial 
recursive function ^: N + -* {0, 1}* such that n < H(Sfr(n)). 
Using ©, we then see that n < H(n) + 0(1) for all n £ N + . 
It follows from (O that n < 21og 2 n + 0(1) for all n £ 
N + . Dividing by n and letting n — > oo we have 1 < 0, a 
contradiction. This completes the proof. ■ 

Theorem 7. Let V be an optimal prefix-free machine, and 
let f: {0,1}* -> N. Suppose that #V _1 (s) < f(s) for all 
s £ {0, 1}* and f is right-computable. Then 

lim {f( S )-W- 1 (s)}=oo. 

s— foo 

Recall here that we identify {0, 1}* with N. 

Proof: We denote by Q the set of all k £ Z such that k < 
/(s)-#U _1 (s) for all but finitely many s £ {0, 1}*. Note that 
£ Q and therefore Q ^ 0. This is because ^V~ 1 {s) < f(s) 
for all s £ {0, 1}*. 



Now, let us assume contrarily that /(s) — ffV 1 (s) does 
not diverge to oo as s — > oo. Then there exists M G N such 
that, for infinitely many s G {0, 1}*, f(s) - #V" 1 (s) < M. 
It is then easy to see that k < M for all k G Q. Thus, since 
Q is a nonempty subset of Z bounded to the above, Q has the 
maximum element fco. Since fco G Q, 

fco < /(«) - #V" 1 («) (8) 

for all but finitely many s G {0, 1}*. If k < f(s) - #V~ l (s) 
for all but finitely many s G {0, 1}*, then ko + 1 G Q and 
this contradicts the fact that fco is the maximum element of Q. 
Thus, fco > f(s) — #V~ 1 (s) for infinitely many s G {0, 1}*. 
Hence, it follows from ([8]) that there exists a finite subset E of 
{0,1}* such that fc < /(s)-#V -1 (s) for all s G {0, 1}*\E 
and fc = /(s)-#V _1 (s) for infinitely many s G {0, 1}*\E. 
We define a function g: {0, 1}* -> N by g(s) = #^ _1 (s) 
if s £ £ and g(s) = /(s) — fco otherwise. Then, obviously, 
#V -1 ( S ) < g(s) for all s G {0,1}* and g is right- 
computable. Moreover, #y _1 (s) = g(s) for infinitely many 
s G {0,1}*. However, this contradicts Theorem [6] and the 
proof is completed. ■ 

Corollary 8. Let V be an optimal prefix-free machine. Sup- 
pose that V~ 1 (s) is a finite set for all s G {0, 1}*. Then the 
function ^V~ 1 (s) of s G {0, 1}* is not bounded to the above. 

Proof: Assume contrarily that the function ^V~ 1 (s) 
of s G {0,1}* is bounded to the above. Then there exists 
M G N such that, for every s G {0,1}*, #V" _1 (s) < M. 
We define a function /: {0,1}* -> N by f(s) = M. 
Then, obviously, #F _1 (s) < f(s) for all s G {0,1}* 
and / is right-computable. It follows from Theorem [7] that 
lim s _ i . 00 {/(s) — #V^ _1 (s)} = oo. However, this contradicts 
the fact that f(s) - W {*) < M for all s e {0, 1}*. This 
completes the proof. ■ 

Theorem 9. For every optimal prefix-free machine V, there 
exists an optimal prefix-free machine W for which the follow- 
ing conditions (i) and (ii) hold: 

(i) H w (s) = H v (s) for every s G {0, 1}*. 
(ii) W~ 1 (s) is an infinite set for every s G {0, 1}*. 

Proof: Let V be an arbitrary optimal prefix-free machine. 
We first show that V^ 1 (sq) has at least two elements for some 
so G {0,1}*- In the case where V~ 1 (sq) is an infinite set 
for some so G {0,1}*, obviously V^ 1 (sq) has at least two 
elements. Thus, we assume that V^ 1 (so) is a finite set for all 
so G {0, 1}*, in what follows. 

First, it follows from Corollary [8] that #V~ 1 (s ) > 2 for 
some s G {0, 1}*. Thus, some V^ 1 (so) has two elements q 
and r with \q\ > \r\. Let b: {0, 1}* x N — > N be a particular 
bijective total recursive function. We then define a partial 
recursive function W: {0,1}* — > {0,1}* by the condition 
that W-^s) = {V- 1 (s)\{q}) U{g0 b < s ^l | i e N} if 
s = so and W (s) = V^ 1 (s) U T(s) otherwise, where 
T{s) = {q0 b( - si h | i G N & H v {s) < \q\ +b(s,i) + l}. 
Since the set { (s,n) G {0, 1}*xN H v (s) < n} is r.e., it 
is easy to see that such a partial recursive function W exists. 



Since b is a bijection, the set { g0 6 ' s ° :i 'l | i G N } is infinite 
and the set T(s) is infinite for every s ^ Sq. Therefore the 
condition (ii) holds for W. On the other hand, it follows that 

domW= (J W _1 (a) 

sG{0,l}* 

c(( (J ^~ 1 (*))\M)u{«o*i|feeN} (9) 

se{o,i}* 

= {domV\{q})U{q0 k l | fcGN}. 

Thus, since doml/ is prefix-free and q G domT^, the most 
right-hand side of (O is prefix-free. Hence its subset dom W 
is also prefix-free, and therefore W is a prefix-free machine. 
Finally, we show that the condition (i) holds for W. In the 
case of s = So, since |g| < |g0 fe l| for all k G N and there is 
r G domV with \r\ < \q\, we have H\v{s) = Hy(s). In the 
case of s ^ sq, since the set T(s) does not contain any string 
of length less than Hv{s), we have Hyv{s) = Hy(s) again. 
Thus, the condition (i) holds for W. ■ 

IV. The Distribution of Codewords 

In this section, we investigate the distribution of codewords 
in V~ 1 (s) for each optimal prefix-free machine V and each 
s G {0,1}*. Solovay [7| showed the following result for the 
distribution of all codewords dom V for an optimal prefix-free 
machine V. 

Theorem 10. Let V be an optimal prefix-free machine. Then 

#{ P G{0,1}* | |p| <nk P £domV} = 2 n - HM+0{1 K 

Namely, there exists d £ N such that 

(i) #{p G {0,1}* | \p\ < n kp G domF} < 2 n - H(n *>+ d 

for all n G N, and 
(ii) ' 2 "- H (")- d < #{pG {0,1}* | \p\ <nkpedomV} 
for all n G N with n — H(n) > d. ■ 

Note that lim n _).oo {n — H(n)} = cxa by J3J. We refine 
Theorem [10] to a certain extent. For that purpose, we define 

S c (n,s) = { P e {0,1}* | |p| <n&C(p) = s} 

for each prefix-free machine C, each n G N, and each s G 
{0, 1}*. We can then show the following theorem. 

Theorem 11. Let C be a prefix-free machine. Then 

#S c (n,s) < 2"- H ("' s )+°M. 

Proof: We show that there exists d G N such that 
#S C (n,s) < 2"- H "( n > s )+ d for all n G N and all s G {0, 1}*. 
For that purpose, we define a function / : N — > [0, oo) by 
f(b(n,s)) = #S , c(n,s)2-"- 1 . Recall here that 6: {0, 1}* x 
{0, 1}* — > {0, 1}* is a particular bijective total recursive func- 
tion. It is easy to see that the set {(a, fc) G Q x N | a < f(k)} 
is r.e. On the other hand, 



E/w= E 



fc=0 



sG{0,l}* "=0 



#Sc(n,s)2" 



= E EE^p,^-"- 1 

sG{0,l}* n =° l =Q 

oo oo 

= e £i>3b(j,»)2- n - 1 

sG{0,l}* Z=0 n=J 

oo 

= E E#^> s ) 2 ~' = E 2-w<i, 

S6{0,1}* Z=0 pSdomC 

where 3c(n,a) = {p G {0,1}* |p| = n & C(p) = 
S }. Thus, / is a lower-computable semi-measure. It fol- 
lows from Theorem |2] that there exists d! G N such that 
f{k) < 2 d '2- ff ( fe ) for all fc G N. Therefore we have 
#5 c (n,s)2- n - 1 < 2 d '- ff ( 6 ( n ' s )) for all n e N and all 
s G {0, 1}*, which implies that #S C (n,s) < 2"- H ( n ^)+ d '+ 1 
for all n G N and all s G {0, 1}*, as desired. ■ 

Theorem Q~3] below shows that the upper bound 
2n-H(n,s)+0{i) m x neorem [JTJ is tight among all optimal 
prefix-free machines. In order to prove Theorems Qj] and [14] 
below, we need the following lemma. 

Lemma 12. H(s) + n — H(H(s) + n, s) diverges to oo as 
n — >• oo uniformly on s G {0, 1}*. Namely, for every Met), 
f/iere exwfs no G N smc/z that, for every n > uq and every 
S G {0, 1}*, H(s) +n- H(H(s) +n,s)> M. 

Proof: Let us consider a prefix-free machine C such 
that, for every p, q G domU, C(pq) = b{\p\ + U(q),U(p)), 
where U(q) is regarded as a natural number based on our 
identification of {0,1}* with N. It is easy to see that such 
a prefix-free machine exists. For each s G {0, 1}* and each 
n G N, we see that C(s*n*) — b(H(s) + n,s) and therefore 
H c {b{H(s) + n,s)) < \s*n*\ = H(s)+H(n). It follows from 
(HJ that there exists d G N such that, for every s G {0, 1}* and 
every n G N, H(b(H(s) + n, s)) < H(s) + H{n) + d. Using 
(O we then see that there exists d' G N such that, for every 
s G {0,1}* and every n G N+, H(s) + n- H(H(s)+n, s) > 
n — 2 log 2 n — dl ' . Hence, the result follows. ■ 

Theorem 13. There exists an optimal prefix-free machine V 
which satisfies that #5y(n,s) = zn-H(n,s)+o(i)_ Name fy 
there exist an optimal prefix-free machine V and d G N such 
that 
(i) #S v (n,s) < 2 n - H{ - n ^+ d for all n G N and all s G 

{0, 1}*, and 
(ii) 2 n - H{n ^- d < #S v (n,s) for all n G N and all s G 
{0, 1}* with n - H(n, s) > d. 

Proof: By Theorem [TT] it is enough to show that the 
condition (ii) holds for some optimal prefix-free machine V 
and some d G N (in fact, d can be chosen to be in the 
following construction of V). 

Let us consider a partial recursive function V: {0,1}* — > 
{0, 1}* such that, for every p, s G {0, 1}*, V(p) = s if and 
only if there exist q, t G {0, 1}* for which p = qt and U(q) = 
b(\p\ , s). Since U is a prefix-free machine and b: {0, 1}* x 
{0, 1}* — > {0, 1}* is a bijective total recursive function, it is 
easy to see that such a partial recursive function V: {0, 1}* — > 



{0, 1}* exists. Since dom U is prefix-free and b is an injective 
function, we can also check that domF is prefix-free. Thus 
V is a prefix-free machine. 

We show that 2 n ~ H{ - n ^ < #S v (n,s) for all n G N and 
all s G {0,1}* with n - H(n,s) > 0. For each n G N 
and t G {0,1}*, if \t\ = n- H(n,s), then \b(n,s)*t\ = n 
and V(b(n,s)H) = s. Recall here that \b(n,s)*\ = H(n,s). 
Thus, for each n G N, if n - H(n, s) > then 2 n - R{ - n ^ < 
#Sv(n,s), as desired. 

Finally, we show that V is optimal. By Lemma [T2l we see 
that there exists no G N such that, for every s G {0, 1}*, 
H(s)+n -H(H(s)+n ,s) > 0. Hence, for each s G {0,1}*, 
\b(H(s) + n ,s)*t\ = H(s) + n Q and therefore V(b(H(s) + 
n ,s)*t) = s, where t = oH(s)+n n -H(H(s)+n ,s)_ Thu ^ we 

see that Hy(s) < -ff (s)+no for all s G {0, 1}*, which implies 
that V is optimal. This completes the proof. ■ 

As a complement to Theorem Qj] the following theorem 
shows that only an optimal prefix-free machine can attain the 
upper bound 2"- ff (™^)+°( 1 ) in TheoremQT] 

Theorem 14. Let C be a prefix-free machine. Suppose that 
2 n-i?(n,s)+0(l) < ^S c {n,s), namely, suppose that there 
exists d G N such that 2 n - H( - n ^~ d < #S C (n,s) for all 
n G N and all s G {0, 1}* with n — H(n, s) > d. Then C is 
optimal. 

Proof: It follows from Lemma[T2lthat there exists no G N 
such that, for every s G {0, 1}*, H(s) + n - H(H(s) + 
uq,s) > d. By the assumption, we see that, for each s G 
{0,1}*, 1 < 2 H ( s H«o--ff(ff(s)+™ , s )-d < #S c (H(s)+n Q ,s). 
Thus, for each s G {0,1}*, S c (H(s) + n ,s) ^ and 
therefore there exists p G {0, 1}* such that \p\ < H(s) + no 
and C(p) = s. Hence, we see that Hc{s) < H(s) + no for 
all s G {0, 1}*, which implies that C is optimal. ■ 
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